Skip to content

Add EQL v3 Drizzle support#565

Open
tobyhede wants to merge 93 commits into
james/cip-3291-bigint-stackfrom
eql-v3-drizzle-concrete-types
Open

Add EQL v3 Drizzle support#565
tobyhede wants to merge 93 commits into
james/cip-3291-bigint-stackfrom
eql-v3-drizzle-concrete-types

Conversation

@tobyhede

@tobyhede tobyhede commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the EQL v3 Drizzle integration (@cipherstash/stack/eql/v3/drizzle) plus hardened live test coverage, on top of the v3 typed client / schema DSL.

  • A Drizzle-native types namespace whose columns are the semantic eql_v3.<domain> Postgres types.
  • createEncryptionOperatorsV3 — capability-checked operators (eq/ne/gt/gte/lt/lte/between/notBetween/contains/inArray/notInArray/asc/desc/and/or/not/isNull/isNotNull/exists/notExists) that emit the two-argument eql_v3 term-function SQL. The concrete column type drives which operators are legal (e.g. contains needs a match index; asc/between need order).
  • extractEncryptionSchemaV3 rebuilds the encrypt schema for EncryptionV3.
  • Live test coverage — a type-driven 35-domain matrix drives live encrypt/decrypt + SQL query proofs. Recent additions strengthen assertion quality: range exclusion (not just inclusion), type-boundary values through real domain columns, NULL persistence across capability tiers, and lock-context querying through the operator path — plus a CI guard that fails loudly if the live suites silently skip.

The existing v2 @cipherstash/stack/drizzle integration is unchanged. v3 free-text search is token containment, so contains replaces SQL like/ilike (deliberately not exposed).

Usage

import { drizzle } from "drizzle-orm/postgres-js"
import { integer, pgTable } from "drizzle-orm/pg-core"
import postgres from "postgres"
import { EncryptionV3 } from "@cipherstash/stack/v3"
import {
  createEncryptionOperatorsV3,
  extractEncryptionSchemaV3,
  types,
} from "@cipherstash/stack/eql/v3/drizzle"

const users = pgTable("users", {
  id: integer("id").primaryKey().generatedAlwaysAsIdentity(),
  email: types.TextSearch("email"),   // free-text search
  nickname: types.TextEq("nickname"), // equality
  age: types.IntegerOrd("age"),       // order + range
})

const client = await EncryptionV3({ schemas: [extractEncryptionSchemaV3(users)] })
const e = createEncryptionOperatorsV3(client)
const db = drizzle({ client: postgres(process.env.DATABASE_URL!) })

// Filter operators are async (they encrypt the query term); asc/desc are sync.
const rows = await db
  .select()
  .from(users)
  .where(await e.contains(users.email, "example.com"))
  .orderBy(e.asc(users.age))

Test Plan

  • cd packages/stack && pnpm biome check src/eql/v3/drizzle __tests__/{drizzle-v3,v3-matrix}
  • cd packages/stack && pnpm vitest run __tests__/{drizzle-v3,v3-matrix}
  • cd packages/stack && pnpm test:types && pnpm build
  • Live suites (Postgres + eql_v3) run when DATABASE_URL + CS_* are set and self-install the eql_v3 extension at beforeAll. The Drizzle lock-context suite additionally needs USER_JWT and soft-skips without it.

Notes

  • Live tests are credential-gated; a new REQUIRE_LIVE / REQUIRE_LIVE_PG guard turns a silent skip into a CI failure so the live matrix can't quietly go green.
  • The two-argument eql_v3.* function forms are interim (tracked by CIP-3402 / CIP-3423).

Summary by CodeRabbit

  • New Features

    • Added support for a new encrypted database integration with richer query capabilities, including equality, ranges, pattern matching, sorting, and membership checks.
    • Introduced a new set of column types for working with encrypted data while keeping query behavior aligned with the underlying field type.
    • Existing encrypted database integration remains available unchanged.
  • Bug Fixes

    • Improved handling of null and undefined values when reading and writing encrypted JSON-backed fields.

tobyhede added 30 commits July 2, 2026 13:10
Column builders are copied onto the EncryptedTable instance for accessor
access (users.email). A column named build/tableName/columnBuilders/
_columnType would silently overwrite that member — worst case a 'build'
column breaks buildEncryptConfig's tb.build() call at runtime.

Throw a clear error at table-definition time instead. Scoped to v3; v2
retains its existing behavior.

Found by CodeRabbit review.
…ntry

The wasm-inline encrypt entry typed opts.column as the widened structural
BuildableColumn, but getColumnName still gated on instanceof EncryptedColumn
|| EncryptedField and threw for a v3 EncryptedTextSearchColumn — a runtime
break the type promise hid. Resolve the name structurally (typeof getName)
so v3 columns round-trip through WasmEncryptionClient.encrypt(); still throws
for non-builder JS input. getColumnName is the only instanceof gate on this
path; the rest reads table.tableName structurally.

Adds wasm-inline-column-name.test.ts exercising the seam (v2 column/field +
v3 column + non-builder). Like its sibling wasm-inline-normalize.test.ts the
suite cannot load in environments missing the @cipherstash/protect-ffi
/wasm-inline dep subpath.
Config tables are keyed by name, so two tables with the same tableName
silently dropped the earlier one. Add a v3-only additive guard that throws
on a duplicate (Object.hasOwn). v2's buildEncryptConfig keeps its existing
silent-overwrite behavior (no-v2-change constraint).
The RESERVED_TABLE_KEYS guard only covered own members (build, tableName,
columnBuilders, _columnType), so a column named constructor/toString/valueOf/
hasOwnProperty was assigned as an own property, shadowing the Object.prototype
member. Add an `in` check (isReservedTableKey) so any prototype-chain member
is also rejected, keeping the table object well-behaved for reflection.
…freeTextSearch' for match

A v3 text_search column emits unique+ore+match, and shared index inference
picks by priority unique > match > ore. So encryptQuery without an explicit
queryType builds an EQUALITY term (via unique) — a substring matches nothing.
Document on EncryptedTextSearchColumn + encryptedTextSearchColumn that callers
must pass queryType:'freeTextSearch' (FFI 'match') for free-text queries.

Addresses review finding #2 (naming footgun; doc-only, no runtime change).
Add table-driven runtime tests for all 40 EQL v3 domain builders (name,
eqlType, capabilities, config, queryability) plus type-level tests for
nominal domain distinctness, InferPlaintext mapping, queryability of
BuildableQueryColumn, and v3/v2 model inference.
Add a generic EncryptedV3Column base parameterised by a literal domain
definition (eqlType, castAs, capabilities), plus concrete builders for every
EQL v3 domain (int2/4/8, float4/8, numeric, date, timestamptz, text*, bool).
Each carries explicit getQueryCapabilities()/isQueryable() metadata, emits
capability-derived indexes, and drives precise InferPlaintext. Refactors
EncryptedTextSearchColumn onto the shared base while preserving its
byte-identical config and match-tuning override.
Tighten BuildableQueryColumn so only capability-bearing, queryable columns
are accepted by encryptQuery. Widen encryptModel/bulkEncryptModels to any
BuildableTable via EncryptedFromBuildableTable (keyed off the _columnType
brand), covering both v2 and v3 tables. Introduce a Plaintext type
(JsPlaintext | Date | bigint) for the single-value encrypt/encryptQuery
entry points so v3 date/timestamptz/int8 domains accept their natural JS
values; cast to JsPlaintext only at the FFI boundary (the FFI declares these
as CastAs targets but omits them from its JsPlaintext input union).
Expand env-gated client and Postgres suites with representative v3 domains
(storage-only, equality, order, match, bigint, date) and a typed-domain PG
table. Add a v3 CJS export assertion. Fix wasm-inline test resolution by
aliasing the unresolvable @cipherstash/{protect-ffi,auth}/wasm-inline
specifiers to local stubs in vitest.config.ts. Includes the v3 install
script, EQL v3 SQL fixture, and eql-v3 test helper.
- types: BuildableTableColumns fallback never -> Record<never, never> so a
  structurally BuildableTable-typed value degrades to the model unchanged
  instead of over-encrypting every field (keyof never is string|number|symbol)
- schema/v3: clone-on-write in EncryptedTextSearchColumn.freeTextSearch so
  caller opts mutated before build() cannot leak into emitted config
- test helper: run EQL v3 advisory lock/check/install/unlock on one reserved
  connection (sql.reserve) instead of across pooled backends
- pg test: clean up protect_ci_v3_typed_domains rows in beforeEach/afterAll
- install script: quiet dotenv config (drop bare dotenv/config banner)
- tests: graceful-degradation, aliased-column, and freeTextSearch clone-on-write
  regression coverage
- docs: replace developer-specific absolute paths with placeholders; wording
…/v3)

Add EncryptionV3 / typedClient returning a TypedEncryptionClient<S> that
derives types from the concrete table/column builder arguments:

- encrypt/encryptQuery pin plaintext to the column domain; encryptQuery
  constrains queryType to the column capabilities and rejects storage-only
  columns at compile time
- encryptModel/bulkEncryptModels validate schema fields against inferred
  plaintext (passthrough fields untouched), precise encrypted output
- decryptModel/bulkDecryptModels return precise plaintext, reconstructing
  Date/bigint from the encrypt-config cast_as

Binding to the concrete branded v3 classes closes the soundness gap where a
non-branded structural table could encrypt at runtime while typed as plaintext.
Runtime is unchanged except a per-column Date/bigint reconstruction on model
decrypt. v2 client surface is untouched.

New @cipherstash/stack/v3 subpath re-exports the v3 builders for a single import.
The native protect-ffi marshals plaintext through JSON serialization,
which throws "Do not know how to serialize a BigInt" for JS bigint
values. int8/bigint v3 domains now accept/return lossless string
plaintext instead; cast_as stays big_int so server-side casting is
unchanged. Fixes the failing schema-v3-client live integration test.

Also hardens adjacent paths surfaced in review:
- encrypt: reject NaN / Infinity number plaintexts
- wasm-inline resolveStrategy: guard missing workspaceCrn/accessKey at
  runtime for JS callers that bypass the compile-time union
- docs: correct query API walkthrough lock-context description

@coderdan coderdan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. A couple of items that should be addressed before merge. Also, the docs are almost non-existent. Should there be a section in the README and some typedoc?

@coderdan

coderdan commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Code review — EQL v3 Drizzle integration + bigint domains

Reviewed at extra-high effort (multi-angle finder pass → verify → sweep) against dbe1ec0. The change is well-tested (deterministic + live-pg matrix), which refuted several plausible bugs (see bottom). 12 findings survived, correctness first.

Correctness / robustness

1. bigint domains are type-blessed but have no runtime guard → opaque low-level errorpackages/stack/src/eql/v3/columns.ts:166 (also types.ts:128, Plaintext in types.ts:96)
types.Bigint/BigintEq/… are exported publicly and PlaintextForColumn resolves them to bigint, but the only value validator (assertValidNumericValue, helpers/validation.ts:43) checks typeof === 'number', so a bigint passes every guard to protect-ffi 0.27, which cannot marshal a JS BigInt across the Neon boundary.
Failure: client.encrypt(123n, { table, column }) on a types.BigintEq column type-checks and passes unit CI (FFI mocked), then fails at runtime with a cryptic Neon/serialization error instead of a clear "bigint not yet supported" SDK error. On the drizzle operator path the throw isn't wrapped in EncryptionOperatorError. The gate is prose comments only — an explicit runtime guard on cast_as === 'bigint' (or not exporting the factories until the pin bumps) would keep the type/runtime contract from diverging silently.

2. decryptModel/bulkDecryptModels key the reconstructor by table object identity → a valid, correctly-typed call can return a failurepackages/stack/src/encryption/v3.ts:217 (Map built at :190)
EncryptedTable is structurally typed (tableName: string, brand is only the phantom _columnType), so two separately-constructed but identical tables are the same type yet different objects. The old code built rowReconstructor(table) from whatever table was passed; the new code does reconstructors.get(table) and returns unknownTableFailure on a miss.
Failure: const c = await EncryptionV3({schemas:[usersA]}) then c.decryptModel(row, usersB) where usersB = encryptedTable('users', {…same…}) (a re-import, HMR reload, or table built in another module). Compiles (structural match), but at runtime returns DecryptionError: "…a table this client was not initialized with…" instead of decrypting — a regression vs. prior behavior. Keying by tableName would match the semantic identity the FFI/build() already use.

3. Drizzle customType declares plaintext data types but toDriver only JSON-serializes (no encryption)packages/stack/src/eql/v3/drizzle/codec.ts:10 (type declared in column.ts:88)
The column is typed data: PlaintextForColumn<C> (Date/bigint/number/string) with toDriver(value) = JSON.stringify(value). The intended flow inserts pre-encrypted envelopes (live tests cast through as never), but nothing in the types enforces that.
Failure: For types.Bigint, the type-valid db.insert(t).values({ n: 1n }) calls JSON.stringify(1n) → throws TypeError: Do not know how to serialize a BigInt synchronously. For date/string, the same type-valid plaintext insert bypasses encryption and JSON-stringifies plaintext into the eql_v3 column (a plaintext-exposure footgun — some domain CHECK constraints may reject the non-envelope shape, but the type actively invites the write). Consider typing data as the encrypted envelope, or documenting/guarding the insert path.

4. Drizzle types mirror can't reach EncryptedTextSearchColumn.freeTextSearch(opts) → tokenization silently locked to defaultspackages/stack/src/eql/v3/drizzle/types.ts:11
Each factory is (name) => makeEqlV3Column(factory(name)), returning a Drizzle column — there's no way to call the chainable .freeTextSearch(opts) tuner. The canonical flow (extractEncryptionSchemaV3(table)EncryptionV3({schemas}), per the live tests) bakes the recovered builder's build().indexes.match into the real encrypt config.
Failure: A drizzle user who needs custom tokenizer/k/m (short-substring or CJK free-text) has no drizzle-path way to set it; every text_search column gets defaultMatchOpts() ngram settings, and contains() behaves against the default tokenization with no error or warning. The JSDoc claims to "mirror" the v3 types namespace but omits this capability.

5. bigint decrypt is unsound once the encrypt gate liftspackages/stack/src/eql/v3/columns.ts:681 (reconstructor at encryption/v3.ts:147)
PlaintextFromKind types bigint columns as bigint, and rowReconstructor intentionally passes them through (only DATE_LIKE_CASTS are rebuilt). The FFI cannot return a JS bigint, so a decrypted bigint column would hold a number/string while typed bigint. Currently masked because encrypt throws first (finding 1).
Failure: After the FFI pin bumps, decryptModel returns the FFI's serialized form for a bigint column but the type asserts bigint; downstream value + 1n throws Cannot mix BigInt and other types. Worth a reconstruction step (or a test) landing alongside the pin bump.

Efficiency

6. inArray/notInArray re-resolve the column context and call build() once per array elementpackages/stack/src/eql/v3/drizzle/operators.ts:269
inArrayOp maps every value through equality(...), and each equality runs resolveContextgetEqlV3Column + builder.build().indexes (:132). The context is identical for all N values.
Failure: inArray(col, [...1000 ids]) triggers 1000 getEqlV3Column calls and 1000 build() invocations (for a text_search column each build() deep-clones the match block). Resolve context once, then reuse it per operand (keep the concurrency-4 encryption). A WeakMap<builder, indexes> memo on build() would also fix per-operator rebuilds.

7. between/notBetween encrypt min then max sequentiallypackages/stack/src/eql/v3/drizzle/operators.ts:237
const encMin = await encryptOperand(...) then const encMax = await encryptOperand(...) — the two operands are independent.
Failure: Every range predicate pays 2× encrypt latency (round-trips to the crypto backend). const [encMin, encMax] = await Promise.all([...]) halves it.

Cleanup / conventions

8. EncryptionOperatorError is redeclared, forking a class that already exists and is exported by v2packages/stack/src/eql/v3/drizzle/operators.ts:29
v2 (src/drizzle/operators.ts:82, re-exported from src/drizzle/index.ts) defines an identical-shaped EncryptionOperatorError. Two distinct runtime class identities: a consumer's catch (e) { if (e instanceof EncryptionOperatorError) } against one import silently fails to match the other, and the shared context shape must be edited in two places. Lift to one shared module.

9. A Linear issue ID appears in a source comment (and in the changeset)packages/stack/src/eql/v3/columns.ts:165 (also .changeset/eql-v3-bigint-domains.md lines 6 and 10)
Internal issue trackers shouldn't leak into this public repo's source or its published CHANGELOG. Drop the IDs or reword. (IDs omitted here for the same reason.)

10. Drizzle table-name extraction via Symbol.for('drizzle:Name') is hand-inlined across ~5 sitespackages/stack/src/eql/v3/drizzle/schema-extraction.ts:11 (again at operators.ts:115; v2 has getDrizzleTableName)
The Drizzle-internal drizzle:Name contract is hard-coded in multiple places with no single source of truth; a Drizzle upgrade that changes it breaks all of them independently. Share one getDrizzleTableName(table) helper.

11. EQL_V3_DOMAINS and buildersByDomain run the same probe iteration twice and can driftpackages/stack/src/eql/v3/drizzle/column.ts:10
Both do Object.values(v3Types).map(f => f('__probe__').getEqlType()), constructing throwaway probe columns per domain. EQL_V3_DOMAINS is exactly buildersByDomain's key set — derive it: new Set(buildersByDomain.keys()).

12. Dead EQL_V3_COLUMN_LEGACY_PARAM (_eqlv3Column) redundancypackages/stack/src/eql/v3/drizzle/column.ts:30
_eqlv3Column is never read outside this file; writeBuilder always sets it together with the Symbol, and readBuilder returns the Symbol first via ??, so the legacy branch is unreachable (the Symbol carrier is confirmed to survive pgTable via config.customTypeParams). The const, type member, extra write, and ?? branch can be deleted; "legacy" also implies a nonexistent external contract.


Cleared during verification (not bugs): eql_v3.eq/neq on ORE-only *_ord columns (live-pg matrix proves exact/complement selection); applyOperationOptions discarding withLock.audit()'s return (audit() mutates this and returns this); resolveMatchOpts vs. the old inline merge (behavior-identical, now safer — it clones for v2 too); EncryptedTextSearchColumn.build() super.build()+spread (byte-identical index block); V3DecryptedModel = V3ModelInput alias (textually identical to the prior mapped type); DATE_LIKE_CASTS filter (same set as === 'date' || === 'timestamp'); new Date(value) reconstruction (correct instant for the UTC samples; unchanged by this PR); mapWithConcurrency (no race, no unhandled-rejection leak); stash recovery before/after pgTable; and the batch-encrypt-query.ts change (type-only, runtime unchanged).

🤖 Generated with Claude Code

tobyhede added 27 commits July 8, 2026 10:31
- Bump stack dependency and the e2e Deno wasm-inline pin to 0.28.0
- Regenerate lockfile
- 0.28 widens JsPlaintext to include bigint (native i64 round-tripping);
  update the v3 type test to assert bigint plaintext is now accepted
- Rename/refresh the FFI changeset for 0.28
…ngeset)

The prior commit f84e4c1 captured only the changeset rename; the actual
content edits were left unstaged. This commit carries them:
- packages/stack/package.json: protect-ffi 0.27.0 → 0.28.0
- e2e/wasm/deno.json: wasm-inline pin → 0.28.0
- pnpm-lock.yaml: regenerated for 0.28.0
- schema-v3.test-d.ts: assert bigint plaintext now accepted (JsPlaintext widened)
- changeset body: 0.27.0 → 0.28.0
…/null)

Close assertion-strength gaps in the EQL v3 live suites where a real
regression could pass green:

- between/notBetween: add narrow single-point range cases that prove
  EXCLUSION (the spanning cases only ever proved inclusion).
- ordering oracle: compare text bytewise, not via localeCompare, to match
  ORE byte order.
- inArray/notInArray: exercise the >4-value concurrency pool.
- matrix-boundary-live-pg: drive catalog boundary samples (INT4/smallint
  bounds, 1e15) through real eql_v3.<type> columns, not just the FFI.
- operators-lock-context-live-pg: query lock-context-bound rows through the
  Drizzle operator path (ops.eq + lockContext/audit); soft-skips on USER_JWT.
- operators-null-live-pg: NULL persistence across storage/eq/ord/match tiers.
- live-gate-required guard + CI: fail loudly (REQUIRE_LIVE/REQUIRE_LIVE_PG)
  instead of silently skipping the live matrices when creds are absent.
Native bigint round-tripping landed in protect-ffi 0.28, so restore the
bigint domains that skip-v3-bigint deferred:

- columns.ts: BIGINT/BIGINT_EQ/BIGINT_ORD/BIGINT_ORD_ORE definitions
  (concrete domain eql_v3.int8* per the SQL fixture, castAs 'bigint'),
  EncryptedBigint*Column classes, AnyEncryptedV3Column union, and
  PlaintextKind/PlaintextFromKind mapping 'bigint' -> JS bigint.
- types.ts / index.ts: types.Bigint* factories and barrel exports.
- v3-matrix/catalog.ts: four int8 rows with i64-bound samples (max/min/0
  and a value past Number.MAX_SAFE_INTEGER); samples widened to bigint.
  Drizzle operator coverage is catalog-driven, so it picks these up too.
- schema-v3.test-d: flip the "bigint not public" negative test to assert
  the factories now exist.
- encryption/v3.ts: bigint needs no decrypt reconstruction (FFI returns a
  real JS bigint on 0.28).
- drop .changeset/skip-v3-bigint.md; add re-enable changeset.

i64 bounds are enforced at the protect-ffi boundary. *_ord_ope is out of
scope (CIP-3403).
@tobyhede tobyhede force-pushed the eql-v3-drizzle-concrete-types branch from 0fa0793 to 17857ee Compare July 8, 2026 04:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants